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DESCRIPTION 
OPERATION GUIDING SYSTEM 

TECHNICAL FIELD 

5 The present invention relates to an operation guiding system which 

guides an operator so that the operator can accurately perform an operation 
composed of a sequence of operation steps. 

BACKGROUND ART 

10 When an operator begins an operation at a manufacturing floor etc., 

he usually begins the operation after he listened to the content of the 
operation from an instructor. 

However, when the operator has to perform an operation composed 
of a sequence of operation steps, he may stop the operation or perform an 

15 improper operation if he runs into an unclear point or has an indistinct 
memory of the content of the operation. Although there is a way that the 
operator performs the operation while listening to the explanation of the 
operation from the instructor, it is inefficient because the instructor always has 
to be around the operator. 

20 

DISCLOSURE OF THE INVENTION 

In view of the above problem, the object of the present invention is to 
provide an operation guiding system which guides an operator so that the 
operator can accurately perform an operation composed of a sequence of 

25 operation steps without an instructor. 

The operation guiding system in accordance with the present 
invention comprises a virtual image memory, a display unit, a virtual image 
replay means, and a virtual image adjusting means. The virtual image 
memory is configured to store, with respect to an operation composed of a 

30 sequence of operation steps, virtual images for explaining a content of each 
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of the operation steps. The display unit is configured to be positioned in front 
of an operator's eye for displaying the virtual images, in contrast to a real 
operation object in front of the operator. The virtual image replay means is 
configured to replay the virtual images on the display unit in order of the 
5 operation steps. The virtual image adjusting means is configured to adjust the 
virtual image such that a virtual operation object drawn on each of the virtual 
images and corresponding to the real operation object will have an 
overlapping relation with the real operation object on the display unit. 

Therefore, the operator can accurately perform an operation 

10 composed of a sequence of operation steps without an instructor, by only 
watching the virtual images on the display unit and performing the operation 
according to the virtual images with respect to the real operation object on 
which the virtual image is overlapped. The operator may begin the operation 
after watching all the virtual images to understand the whole operation. 

15 Because this operation guiding system includes the virtual image adjusting 
means, the virtual operation object drawn on the virtual image can keep the 
overlapping relation with the real operation object on the display regardless of 
the operator's position or the height of the operator's eye. Therefore, the 
operator can perform the operation efficiently. 

20 Preferably, the virtual image includes a line drawing which outlines 

the real operation object and a visual information mark for visually explaining 
the content of the operation step. The line drawing facilitates visualization of 
the real operation object as well as the virtual operation object when they 
overlap each other, and the visual information mark can visually and easily 

25 convey the information that the operator should know for the operation, to the 
operator. So, the operator can perform the operation more accurately. 

Furthermore, the operation guiding system preferably includes a 
visual information mark input means which calls up the virtual image from the 
virtual image memory to modify the visual information mark thereof and/or 

30 add the visual information mark thereto. Including the visual information mark 
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input means enables the system to change flexibly, even if an operation 
direction is changed or new information that the operator should know arises. 
In this case, it is preferable that the virtual image comprises a layer on which 
the line drawing is drawn and a layer on which the visual information mark is 
5 drawn. Such virtual image can prevent an accidental deletion of the 
complicated line drawing and can provide a worry-free modification and/or a 
worry-free addition of the visual information mark. 

Preferably, the virtual image replay means memorizes a 
correspondence between the virtual image and the operation step and has a 

10 function that calls up the virtual image corresponding to the operation step 
specified by the operator. Such function enables the operator to replay the 
virtual image that he needs quickly, even if the operation is composed of 
many operation steps. 

It is also preferable that the virtual image replay means memorizes a 

15 correspondence between the virtual image and the operation step and has a 
function that replays the virtual images corresponding to the steps within a 
range specified by the operator and returns to the first virtual image in the 
range after the replay. In this case, the operator can watch the virtual images 
in a range that he needs in order to understand the whole operation, and then 

20 he can start the operation quickly in keeping with the replay of the virtual 
image because the virtual image has returned to the first step in the specified 
range after the replay. 

Furthermore, the operation guiding system preferably includes a voice 
input means for inputting the operator's voice and a voice recognition means 

25 that recognizes the voice inputted using the voice input means. The virtual 
image replay means controls the replay of the virtual images based on the 
operator's voice. In this case, the operator can control the replay of the virtual 
images with hands full of tools, etc., so the operation efficiency can be 
improved more. 

30 Furthermore, the operation guiding system preferably includes a voice 
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memory for storing voice data which explains the content of the operation 
steps, and a voice output means for outputting the voice data. The virtual 
image replay means outputs the voice data to the voice output means in 
synchronization with a replay of the virtual images. In this case, the operator 
5 can receive the voice explanation about the operation in addition to the virtual 
image, so that he can understand the content of the operation more 
accurately. The voice data can explain the information which the virtual image 
can not be expressed. 

The virtual image adjusting means may include an image pickup 

10 means for taking an image of the real operation object and a feature point 
extraction means for extracting a feature point decided in advance with 
respect to the operation object, from the image taken by the image pickup 
means, and, the virtual image adjusting means may change a position and/or 
a dimension of the virtual image displayed on the display unit automatically so 

15 that a position of a point of the virtual operation object which corresponds to 
the feature point extracted by the feature point extraction means will conform 
to the position of the feature point. In this case, even if the operator changes 
his position during the replay of the virtual image, the virtual image is adjusted 
automatically, and the virtual operation object can keep the overlapping 

20 relation with the real operation object. 

Alternatively, the virtual image adjusting means may include a manual 
controller by which the operator can change the position and/or the dimension 
of the virtual image displayed on the display unit manually. In this case, the 
operator can bring the virtual operation object into the overlapping relation 

25 with the real operation object by hand. In this case, the operation guiding 
system preferably includes a head tracking means for tracking a motion of the 
operator's head, and the virtual image adjusting means corrects the position 
of the virtual image displayed on the display unit based on the output of the 
head tracking means. By including the head tracking means, once the 

30 operator brings the virtual operation object into the overlapping relation with 
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the real operation object by the manual controller, the virtual operation object 
in the virtual image can keep the overlapping relation with the real operation 
object even if the operator moves his head. 

5 BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram of an operation guiding system in accordance with a 
first embodiment of the present invention. 

FIGS. 2A, 2B, and 2C are examples of the virtual images used in the 

operation guiding system. 
10 FIG. 3 is a picture, as a substitute for a drawing, obtained by printing an 

image showing feature points of an engine as an operation object. 

FIGS. 4A and 4B are pictures, as substitutes for drawings, obtained by 

printing images displayed on a display unit of the operation guiding system. 

FIG. 5 is a picture, as a substitute for a drawing, obtained by printing an 
15 image displayed on a display unit of the operation guiding system. 

FIG. 6 is a picture, as a substitute for a drawing, obtained by printing an 

image displayed on a display unit of the operation guiding system. 

FIG. 7 is a diagram showing another construction of a glass section of the 

operation guiding system. 
20 FIG. 8 is a block diagram of the operation guiding system in accordance with 

a second embodiment of the present invention. 

FIG. 9 is a picture, as a substitute for a drawing, obtained by printing an 
image displayed on a display unit of the operation guiding system. 
FIG. 10 is a picture, as a substitute for a drawing, obtained by printing an 
25 image displayed on a display unit of the operation guiding system. 

FIG. 11 is a picture, as a substitute for a drawing, obtained by printing an 
image displayed on a display unit of the operation guiding system. 

BEST MODE FOR CARRYING OUT THE INVENTION 
30 Hereinafter, the present invention will be described in more detail with 
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reference to the accompanying drawings. 

FIG. 1 shows an operation guiding system 1 in accordance with a first 
embodiment of the present invention. In this embodiment, the operation 
guiding system 1 is configured to be used in a manufacturing floor in which an 
5 engine is assembled, and it leads an operator so that he or she can 
accurately perform an engine assembly operation. 

The operation guiding system 1 includes a head mounted AV (audio 
and visual) unit 10 and an operation guiding control unit 15 connected 
electrically to the AV unit 10 through a cable C1 . 

10 The AV unit 10 is composed of an integrated combination of a glass 

section 1 1 as a display unit, a headphone section 12 as a voice output means, 
a microphone section 13 as a voice input means, and a CCD camera 100 as 
an image pickup means. The glass section 1 1 is like a so-called see-through 
head mounted display, and it has a pair of translucent liquid crystal panels 

15 111. The headphone section 12 has a pair of headphone speakers 121. The 
microphone section 13 has a microphone 131. The CCD camera 100 takes 
an image of the engine as an operation object from the operator's line of sight. 
The operator can see the real object through the translucent liquid crystal 
panels 111 in a state where a virtual image, as described later, is displayed 

20 on the crystal panels 1 1 1 . In other words, the glass section 1 1 can display the 
virtual image, in contrast to a real image. 

The operation guiding control unit 15 comprises an interface section 
16, a memory section 17, a power supply 18, and a control section 19. 

The interface section 16 includes a voice input interface 161, an 

25 image input interface 162, a virtual image output interface 163, and a voice 
output interface 164. The voice input interface 161 amplifies an analog voice 
signal from the microphone 131 and converts it into a digital voice signal and 
outputs to the control section 19. The image input interface 162 converts an 
analog image signal form the CCD camera 100 into a digital image data and 

30 outputs to the control section 19. The virtual image output interface 163 is 
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based on the DVI (Digital Visual Interface) standard, and it connects between 
the control section 19 and the liquid crystal panels 111. The voice output 
interface 164 converts a voice data from the control section 19 into an analog 
voice signal and amplifies it based on a gain corresponding to a position of a 
5 volume control (not shown) and outputs to the headphone speakers 121 . 

The memory section 17 is a storage device such as a hard disk, and it 
includes a virtual image memory segment 17A and a voice memory segment 
17B. The virtual image memory segment 17A stores, with respect to the 
assembly operation of the engine composed of a sequence of a plurality of 

10 operation steps, the virtual images for explaining a content of each of the 
operation steps. Each of the virtual images is created based on a line drawing 
that draws an outline (in other words, an outer shape) of the operation object, 
and a visual information mark for explaining the content visually is added 
thereto as needed. The visual information mark is, for example, a number 

15 showing an operation order, an arrow showing a direction in which the 
operation is performed, text information, a completion drawing of the 
operation shown by dashed lines, and so on. FIGS. 2A, 2B, and 2C show 
examples of the virtual images. They are for explaining one operation of the 
engine assembly operation in which the operator fastens a bolt to an engine 

20 block and then attaches an engine cover thereto. FIG. 2A is a virtual image in 
which an outline of the engine block before the operation is drawn. FIG. 2B is 
a virtual image corresponding to a step in which the operator will fasten the 
bolts. FIG. 2C is a virtual image corresponding to a step in which the operator 
will attach the engine cover. Referring to FIG. 2B for example, the virtual 

25 image is created based of a line drawing that draws an outline of the engine 
as the operation object, and visual information marks 11 to 113, composed of 
numbers and text information, are added thereto. The visual information 
marks 11 to 112 show the order in which the operator is supposed to fasten the 
bolts, and the visual information mark 113 shows tightening torque of the bolts. 

30 The voice memory segment 1 7B stores voice data for explaining the content 



8 



of each of the operation steps, such as an operation sequence, information 
associated with Product Liability Law, cautions, and so on. For example, the 
voice data replayed in synchronization with FIG. 2B is as follow: "In this step, 
you are supposed to fasten bolts in the order from 1 to 6. At that time, the 
tightening torque is 19.5 ± 2.0 Nm". By using the voice data and the visual 
information mark appropriately, the operator can understand the operation 
content more easily and can perform the operation accurately. It is preferable 
that an explanation which the operator can understand if he listens once (for 
example, an outline of the operation) is created as voice data and an 
explanation which the operator can not understand if he listens to only once 
(for example, an operation sequence, detailed numerical value, etc.) is 
created as a visual information mark. 

The control section 19 is configured to control the whole operation 
guiding system 1, and it comprises a CPU (Central Processing Unit) for 
performing a predetermined program, main memory, and so on. The control 
section 19 has a voice recognition function 191, a feature point extraction 
function 192, a virtual image control function 193, and a voice output function 
194. 

The voice recognition function 191 recognizes the voice data, which is 
inputted from the microphone 131 and taken in through the voice input 
interface 161, and controls a virtual image replay function 193A according to 
the recognized voice. For example, when the voice recognition function 191 
recognizes the term "Start replay", it issues an instruction to the virtual image 
replay function 193A to start the replay of the virtual image. 

The feature point extraction function 192 extracts feature points 
decided in advance with respect to the engine as the operation object, from 
the image that is taken by the CCD camera 100 and taken in through the 
image input interface 162. For example, as shown in FIG. 3, if the feature 
points of the engine has been decided in advance to a point P1 and a point 
P2, the feature point extraction function 192 extracts the points P1, P2 from 
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the image taken by the CCD camera, and it determines the positions of the 
pixels of the feature points P1 , P2 in the image. 

The virtual image control function 193 further includes the virtual 
image replay function 193A and a virtual image adjusting function 193B. The 
5 virtual image replay function 193A calls up the virtual images from the virtual 
image memory segment 17A to display them on the liquid crystal panels 111 
in the order of the operation steps of the engine assembly operation. 
Furthermore, the virtual image replay function 193A memorizes a 
correspondence between each of the virtual images and each of the 

10 operation steps, and it has a so-called "cue function" that calls up the virtual 
image corresponding to one of the operation steps that the operator specified 
using the voice recognition function 191, and it also has a "specified range 
replay function" that replays the virtual images corresponding respectively to 
the operation steps within a range specified by the operator and returns to the 

15 first virtual image in the specified range after the replay of the virtual images. 
Of course, the virtual image replay function 193A can fast-forward, rewind, 
and pause the virtual image(s). The virtual image adjusting function 193B 
adjusts the virtual image replayed by the virtual image replay function 193A 
so that the line drawing of the engine drawn on each of the virtual images will 

20 have an overlapping relation with the real engine as the real operation object 
which can be seen through the glass section 1 1 . The adjusting method will be 
explained below. First, a table that associates the position of each of the 
pixels of the image data taken by the CCD camera 1 00 with the position of 
each of the pixels of the liquid crystal panel 1 1 1 is created. The table is stored, 

25 for example, in ROM (not shown) in the control section 19. The virtual image 
adjusting function 193B converts the position, in the image data, of each of 
the pixels of the feature points P1 , P2 extracted from the image data by the 
feature point extraction function 192, into the position of the corresponding 
pixel in the liquid crystal panels 111, with reference to the above-mentioned 

30 table. Then, the virtual image adjusting function 193B changes the position 
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and/or the dimension of the virtual image automatically so that feature points 
A, B in the virtual image displayed on the liquid crystal panels 111 that 
corresponds to the feature points P1, P2 respectively will conform to the 
positions of the pixels of the feature points P1 , P2 respectively on the liquid 
5 crystal panels 111. FIG. 4A shows a real engine and a virtual engine 
displayed on the liquid crystal panels 111 before the virtual image adjusting 
function 193B adjusts the virtual image. In FIG. 4A, because of the operator's 
position, etc., the line drawing of the engine drawn on the virtual image is 
displayed on the liquid crystal panels 111 in a misalignment relation with the 

10 real engine seen through the liquid crystal panels 111. FIG. 4B shows the 
state after the virtual image adjusting function 193B adjusts the virtual image. 
In FIG. 4B, the line drawing of the engine drawn on the virtual image is 
displayed in the overlapping relation with the real engine by the virtual image 
adjusting function 193. The adjustment of the virtual image by the virtual 

15 image adjusting function 193 is executed repeatedly in real time. 

The voice output function 1 94 memorizes a correspondence between 
each of the virtual images and each piece of the voice data stored in the voice 
memory segment 1 7B, and it outputs a piece of the voice data corresponding 
to the virtual image replayed by the virtual image control replay function 193A 

20 to the voice output interface 164 in synchronization with the replay of the 
virtual image. 

Now, the operation guiding system 1 constituted as mentioned above 
is used as follows. In what follows, for the sake of understanding, the 
assembly operation of the engine is roughly classified into a "step 1", a "step 
25 2", and the operator is to perform the operation called the "step 2". The 
step 2 further comprises a "step 2a" and a "step 2b". In the step 2a, the 
operator is supposed to fasten bolts to the engine block, and in the step 2b, 
the operator is supposed to attach an engine cover to the engine block after 
the step 2a. 

30 First, the operator puts on the AV unit 10 and sees the 
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unaccomplished engine through the liquid crystal panels 111, as shown in 
FIG. 3. When the operator says, "Step 2", toward the microphone 131, the 
voice recognition function 191 recognizes the term "step 2", and then the 
virtual image replay function 193A calls up one virtual image which is a pre- 
operation image of the step 2 (that is, FIG. 2A.) on the liquid crystal panels 
111, using the "cue function". At that moment, the virtual image adjusting 
function 193 adjusts the virtual image, so that the virtual image is displayed 
on the liquid crystal panels 111 in a state where the line drawing of the engine 
entirely overlaps with the real engine, as shown in FIG. 4B. Next, when the 
operator says, "Start the replay", the virtual image replay function 193A 
replays the virtual image corresponding to the step 2a (that is, FIG. 2B) in 
accordance with the order of the assembly operation (the step 2a - the step 
2b). And the voice explanation is replayed from the headphone speakers 121 
by the voice output function 194. FIG. 5 shows a view looked through the 
liquid crystal panels 1 1 1 at that time. The operator watches the display of the 
liquid crystal panel while listening to the voice explanation, and he fastens the 
bolts to the engine block in accordance with the virtual images. When the 
operator has finished fastening the bolts, he says, "Next operation", and then, 
the voice recognition function 191 recognizes the term "next operation", the 
virtual image replay function 193A replays the virtual image corresponding to 
the next step, i.e., the "step 2b" (that is, FIG. 2C) on the liquid crystal panels 
111. FIG. 6 shows a view looked through the liquid crystal panels 111 at the 
time. Then, the operator attaches the engine cover to the engine block, while 
watching the display of the liquid crystal panel and listening to the voice 
explanation again. 

The operator can rewind, pause, or fast-forward the virtual images as 
needed. The operator may first replay all the virtual images corresponding to 
the step 2 (that is, FIG. 2A - FIG. 2B - FIG. 2C) continuously using the above- 
mentioned "specified range replay function" to watch the whole operation, like 
watching animation, and to understand the whole operation, and then, he may 
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begin the operation. In this case, because the virtual image displayed on the 
liquid crystal panel will return to the first step in the step 2 (that is, FIG. 2A) 
after the operator watched the whole virtual images, the operator can easily 
start the replay of the virtual images from FIG. 2A again and perform the 
5 operation in keeping with the replay of the virtual image. The operator can 
control the replay of the virtual image with hands full of tools, by the voice 
recognition function 191 . Even if the operator moves his head in the middle of 
the operation, the virtual image adjusting function 193B adjusts the virtual 
image so that the line drawing of the engine drawn on the virtual image will 

10 keep the overlapping relation with the real engine. As explained above, this 
operation guiding system 1 enables the operator to accurately perform the 
operation, without an instructor. 

Although the memory section 17 is provided within the operation 
guiding control unit 15 in this embodiment, the memory section 17 may be 

15 provided in a server connected to the operation guiding control unit 15 by a 
wireless or wired LAN. In this case, the operation guiding control unit 15 has 
a wireless or wired LAN interface. The memory 17 is not limited to the hard 
disk, but it may be, for example, an optical disk device, a magnetic optical 
disk device, or a semiconductor memory device. 

20 Although the above glass section 1 1 was the translucent liquid crystal 

panels 111, it may use a prism shown in FIG. 7. In FIG. 7, the virtual image 
on the liquid crystal panels 111 enters the prism 112 from one side thereof, 
and it is totally reflected forward by a reflecting surface 112a on the rear 
surface of the prism 112, and then it is reflected backward by a half mirror 

25 1 12b on the front surface of the prism 112, and then it transmits the reflecting 
surface 112a. As a consequence, the operator can watch the virtual image. 
On the other hand, the operator can see the operation object itself by an 
optical path penetrating from the front of the prism 1 1 2 to the rear thereof. 
When the background color of the virtual image is set in black around 

30 "000000" (web safe colors) or dark gray around "777777" and the line drawing 
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color of the virtual image is set in light blue around "OOFFFF", the virtual 
image becomes very clear, in contrast to the operation object itself. Similarly, 
when the color of the character and the symbol is set in light blue around 
"OOFFFF" or yellow around "FFFFOO", they become very clear. The glass 
5 section 1 1 does not need to have two panels, but it may have one panel. That 
is, the display unit of the present invention is not limited to a specific one, but 
it has only to display the virtual image for the operator together with the real 
operation object in front of the operator. 

The virtual image adjusting function 193 does not need to adjust the 

10 virtual image in real time, but it may adjust it at specified time intervals. 

The "specified range replay function" can replay the virtual images 
over two or more steps, of course. 

It is preferable that the system further includes a visual information 
mark input means (not shown) that calls up the virtual image from the virtual 

15 image memory segment 1 7A to modify the visual information mark included in 
the virtual image and/or to add new visual information mark to the virtual 
image. The visual information mark input means enables the system to 
respond flexibly to a minimal change of the operation generated at the shop- 
floor level (for example, a change of a bolting order), so that the operator can 

20 perform the operation more effectively. In this case, it is preferable that the 
virtual image is created separately divided into a layer on which the line 
drawing outlining the real operation object is drawn and a layer on which the 
visual information mark is drawn in order to prevent an accidental deletion of 
the complicated line drawing and to provide a worry-free modification and/or a 

25 worry-free addition of the visual information mark. 

Although the engine was taken as an example of the operation object 
in this embodiment, the operation object is not limited to a "thing" like the 
engine, but it may be a man or an animal. For example, this operation guiding 
system can be used for teaching someone how to put on the kimono or for 

30 checking the procedure of an operation on a man or an animal. 
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FIG. 9 shows an operation guiding system 2 in accordance with a 
second embodiment of the present invention. The basic composition of the 
system 2 is identical to the first embodiment except that the virtual image 
5 adjusting function 193B adjusts the virtual image according to a manual 
intervention. The operation guiding system 2 newly includes a handheld 
mouse 20 as a manual controller, which is connected to the operation guiding 
control unit 15. The AV unit 10 includes a head tracker 200 as a head tracking 
means in the glass section 11 instead of the CCD camera 100. The head 

10 tracker 200 tracks a motion of the operator's head. The operation guiding 
control unit 15 newly includes a manual controller interface 165 and a head 
tracker interface 166 in the interface section 16. The manual controller 
interface 165 is connected to the handheld mouse 20, and the head tracker 
interface 166 is connected to the head tracker 200. The image input interface 

15 162 has been removed from the interface section 16, and the feature point 
extraction function 192 has been removed from the control section 19. 

Now, the method for adjusting the virtual image by the virtual image 
adjusting function 193B will be described below. The control section 19 has a 
virtual image replay mode and a virtual image adjusting mode, and they are 

20 switched to each other by the voice recognition function 191 or the handheld 
mouse 20. In the virtual image adjusting mode, a feature point decided in 
advance with respect to the virtual operation object of the virtual image is 
displayed on the liquid crystal panels 111 in addition to the real operation 
object and the virtual image. For example, in FIG. 9, feature points A, B of the 

25 engine of the virtual image are displayed in addition to the real engine and the 
virtual engine. In FIG. 9, the real engine and the virtual engine are displayed 
in a misalignment relation. In such a state, the operator drags the position of 
the feature point A to a corresponding feature point P1 of the real engine 
seen through the liquid crystal panes 111, using the handheld mouse 20. At 

30 this time, the virtual image adjusting function 193B moves the whole virtual 



15 



image according to the drag operation. When the operator stops the dragging, 
the position of the feature point A is fixed (see FIG. 10). Next, the operator 
drags the position of the feature point B to a corresponding feature point P2 
of the real engine. At this time, the virtual image adjusting function 193B 
5 changes the size of the whole virtual image with reference to the position of 
the feature point A. When the operator has finished above mentioned 
operation, the real engine and the virtual engine of the virtual image are 
displayed in the overlapping relation, as shown in FIG. 1 1 . 

The adjustment executed in the virtual image adjusting mode is 

10 applied to all the virtual images stored in the memory section 17, so once the 
operator adjusts the virtual image in the virtual image adjusting mode, the 
operator does not need to adjust the virtual image every time the virtual 
image changes in the subsequent virtual image replay mode. 

Even if the operator moves his head in the virtual image replay mode, 

15 the head tracker 200 corrects the position of the virtual image automatically, 
so the virtual image can keep the overlapping relation with the real operation 
object. Concretely speaking, if the head tracker detects that the operator's 
head moves to the left, the virtual image adjusting means 193B moves the 
virtual image to the right as much as the head moves. As the head tracker, an 

20 inertial measurement unit which applies an accelerometer or a gyro for the 
head mounted display can be used. 

The method for adjusting the virtual image is not limited to the above- 
mentioned method. For example, the control section 19 may move 
automatically into a size change mode after the operator has decided the 

25 position of the feature point A. In the size change mode, the size of the whole 
virtual image is changed with reference to the position of the feature point A 
according to the moving amount and moving direction of the handheld mouse 
20. The virtual image may be able to be adjusted any time except when it is 
being replayed, without an operator's switching operation from the virtual 

30 image reproducing mode to the virtual image adjusting mode. 
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Instead of the handheld mouse 20, a joy-stick type manual controller 
or various kinds of pointing device can be used. 



